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Abstract 

Background: Simulation applications on operations management in hospitals are frequently published and claim 
to support decision-making on operations management subjects. However, the reported implementation rates of 
recommendations are low and the actual impact of the changes recommended by the modeler has hardly been 
examined. This paper examines: 1) the execution rate of simulation study recommendations, 2) the research 
methods used to evaluate implementation of recommendations, 3) factors contributing to implementation, and 4) 
the differences regarding implementation between literature and practice. 

Results: Altogether 16 hospitals executed the recommendations (at least partially). Implementation results were 
hardly reported upon; 1 study described a before-and-after design, 2 a partial before and after design. Factors that 
help implementation were grouped according to 1) technical quality, of which data availability, validation/ 
verification with historic data/expert opinion, and the development of the conceptual model were mentioned 
most frequently 2) process quality, with client involvement and 3) outcome quality with, presentation of results. 
The survey response rate of traceable authors was 61%, 18 authors implemented the results at least partially. 
Among these responses, evaluation methods were relatively better with 3 time series designs and 2 before-and- 
after designs. 

Conclusions: Although underreported in literature, implementation of recommendations seems limited; this review 
provides recommendations on project design, implementation conditions and evaluation methods to increase 
implementation. 

Methods: A literature review in PubMed and Business Source Elite on stochastic simulation applications on 
operations management in individual hospitals published between 1997 and 2008. From those reporting 
implementation, cross references were added. In total, 89 papers were included. A scoring list was used for data 
extraction. Two reviewers evaluated each paper separately; in case of discrepancies, they jointly determined the 
scores. The findings were validated with a survey to the original authors. 

Keywords: Simulation, Operations management. Implementation, Process improvement. Hospitals 



Background 

The median spending on healthcare in the Organisation 
for Economic Co-operation and Development (OECD) 
countries is 8.8% of GDP, with the USA spending 15% 
[1]. These countries are struggling to contain costs, for- 
cing hospitals to rethink strategies on efficiency and the 
organization of processes. Due to the complexity and 
variability of many processes, managers find it difficult 
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to estimate whether a redesign will result in significant 
improvements. To overcome this, hospitals need techni- 
ques that support them in making well-informed deci- 
sions about the trade-off between costs and quality [2]. 
Simulation provides various techniques that help hospi- 
tals face these challenges [3,4]. 

The perceived advantages of simulations and the 
growing number of applications [5] suggest this is a sui- 
table approach for the healthcare sector. After reviewing 
163 papers on operations research (OR) in healthcare 
Brailsford et al.[6] concluded that simulation is the 
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second most popular OR technique after statistical ana- 
lysis. These results are comparable with other sectors 
[7,8]. 

The implementation of simulation recommendations 

In the past 30 years, at least seven reviews on simulation 
in healthcare appeared [5,6,9-13], leading to the conclu- 
sion that simulation is widely used and can be regarded 
as a mature tool [14]. Four papers researched the preva- 
lence of implementing recommendations derived from 
OR models in healthcare [6,9,13,15]. 

After reviewing more than 200 papers on simulation, 
Wilson [13] found only 16 studies reporting the execu- 
tion of recommendations and only 11 of these papers 
described operational problems in healthcare. Lagergren 
found that almost two-thirds of the papers on OR models 
in healthcare discussed general OR aspects or did not 
report on execution [15]. In a review of 182 papers about 
simulation modeling in population health and health care 
delivery, Fone et al. [9] concluded that evidence of imple- 
mentation is scarce. More recently Brailsford et al. [6] 
presented an extensive literature review of different types 
of modeling efforts in healthcare. They examined imple- 
mentation on a three level scale: suggested (theoretically 
proposed by authors), conceptualized, implemented 
(actually used in practice). Only 5.3% of the 342 papers 
reported to have been used in practice. 

Although these reviews provided insight into the pre- 
valence of implementing simulation recommendations, 
it remains unclear if the results are valid in the present 
context. Wilson's study [13] although comprehensive, 
was completed in 1981, which leads to the question 
whether the conclusions are still valid, given the 
advances in simulation software and techniques. The 
review of Lagergren was not focused solely on simula- 
tion and was by the author's own account "incomplete" 
[15]. Fone et al. [9] reviewed simulations on population 
health and healthcare delivery instead of operations 
management in individual hospitals. Although the work 
of Brailsford et al. [6] appeared comprehensive, the 
results were not limited to simulation and did not 
examine the realized impact of the changes recom- 
mended by the simulation study. 

We conclude that the reported implementation rates 
are low, and that none of the mentioned reviews exam- 
ined the realized impact of the changes recommended 
by the modeler. This suggests that although simulation 
is widely reported upon in healthcare, it is not clear 
whether actual implementation is carried out by 
management. 

Realizing improvements with simulation models 

To achieve improvements with simulation studies, one 
needs both a competent change management strategy 



and a simulation model and results deemed acceptable 
by the stakeholders. This paper focuses on simulation 
models. For references on change management, suffi- 
cient papers are available [16-19]. 

We identified two frameworks on the development of 
simulation models in healthcare [20,21]. Both empha- 
sized that in healthcare the problem definition phase 
consumes more time due to conflicting stakeholder 
objectives and an unclear problem understanding. Addi- 
tionally, more involvement of staff is required because 
decision makers are often unfamiliar with simulation 
techniques and therefore treat it with suspicion. 

Understanding the relation between simulation models 
and improvements requires insight into the conditions 
that increase the implementation rate of recommenda- 
tions. To our knowledge, four papers [22-25] have 
strived to identify these conditions. However, the focus 
and results were not specific to the healthcare sector. 
Robinson and Pidd interviewed 10 simulation modelers 
and 10 organizations to determine factors considered as 
important [24]. This led to SIMQUAL [25], a survey 
that compared the expected quality with the perceived 
quality. McHaney and Cronan used a contingency 
model of simulation success as input for a survey 
among 126 projects to examine the relation between 
simulation project characteristics and their success in 
126 projects [23]. 

Research objectives 

Several research gaps exist on the relation between 
simulation applications in healthcare and the execution 
of the recommendations. Therefore, we report on the 
following research objectives: 

1. To determine the frequency that simulation recom- 
mendations are executed to improve operations manage- 
ment in individual hospitals. 

2. To determine what factors contribute to the imple- 
mentation of simulation study recommendations. 

3. To determine the research methods used to evalu- 
ate implemented simulation recommendations. 

4. To examine the difference between literature and 
reality with regard to the implementation of simulation 
recommendations. 

The answers to these questions can support the trans- 
formation from simulated scenarios to improved hospi- 
tal processes. 

Results 

The literature search strategy resulted in 161 abstracts 
in PubMed and 125 in Business Source Elite (BSE), in 
total 277 different abstracts. The reviewers selected 113 
abstracts for inclusion. We obtained the full text of all, 
except 2 papers. Only 68 met all inclusion criteria. The 
cross reference check on those partially implemented 
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simulation recommendations, resulted in 21 additional 
papers. In total, 89 papers were included. Figure 1 visua- 
lizes the paper selection process. The methods section 
in the end of this manuscript describes the details of the 
search strategy and the inclusion process. 

Section I: Project scope and background 

Altogether, 68 papers simulated a single department, 21 
multiple departments. Of all multi-department simula- 
tions, 12 included the nursing wards, 11 the operating 
theatre, 7 the emergency department, 7 diagnostic facil- 
ities, 3 the intensive care and 1 the pharmacy. The most 
frequently examined single departments are the emer- 
gency & accidents department (18 papers), the opera- 
tions theatre (15 papers) and the consultations 
department (13 papers). Additionally, 9 papers reported 
the use of their model in more than one setting. 

Section II: Implementation phases 

Table 1 shows that the simulation models of 73 papers 
presented (partial) direct benefits to the hospital and 26 
stated that the hospital (partially) accepted their results. 
Only 10 papers reported the execution of recommenda- 
tions, while 6 reported partial execution (totaling to 
18%) and 3 mentioned the intention to do this. 

Section III: Assessing the evidence that simulation leads 
to improvements 

Only 3 papers reported data on the effects of the imple- 
mentation. One of these papers had a before and after 
design (for definition see [26]) and 2 papers described a 
few results after the implementation. The post 



implementation measurements showed that the model of 
1 paper was correct, whereas the other 2 papers were cor- 
rect on most of the evaluated aspects. 

Section IV: Technical quality factors of a simulation study 

Table 2 shows data availability is the most frequently 
mentioned factor that contributes to implementation of 
recommendations (57 times). The validation and verifi- 
cation of the simulation model through historic data 
was also identified as important (37 times). The same is 
true for validation and verification through expert opi- 
nion (30 times). Furthermore, the quality of the concep- 
tual model was mentioned in 31 papers. The others' 
category mentioned the choice for modeling software 
and user friendliness 5 times, and modeler skills, 4 
times. 

Section V: Factors related to the process quality of a 
simulation study 

Table 2 also presents the process quality factors. Client 
involvement was the most frequently mentioned factor 
(21 times), followed by appropriate use of animation (19 
times). In the others' category, 5 papers reported the 
importance of allowing sufficient time for the hospital 
to experiment with the model. 

Section VI: Factors related to the outcome quality of a 
simulation study related to implementation as stated by 
the authors 

Most often (15 times), the presentation of the results 
was mentioned. Negative factors were: the simulated 
recommendations show improvements on one aspect 



161 abstracts from PubMed 



125 abstracts from BSE 



277 different abstracts 



2 papers 
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70 relevant papers 
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references of 
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literature review 



Figure 1 Overview selected papers for literature review. 
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Table 1 Results Section II: implementation phases 



Implementation phases 


Yes 


Partially 


Intention is mentioned 


No 


Wot stated 


Did tine study achieve the clients objectives 


89 


0 


N/A 


0 


0 


Show the study results direct benefits to the client? 


71 


2 


N/A 


11 


5 


The study results are accepted by the client 


21 


5 


N/A 


0 


63 


The study results are executed 


10 


6 


3 


1 


69 



but deterioration on others (10 times), changing circum- 
stances during the project (5 times) and doubts on the 
cost benefit ratio (3 papers). In the others' category, 4 
papers stated that, although relevant for implementation, 
a simulation model cannot include cultural or behavioral 
aspects. 

Survey 

After a reminder we received responses for 41 papers; 
29 returned the survey; for 12 papers the authors 
responded that they did not have the intention to 
improve a specific hospital process, the data only served 
to illustrate the potential of the model. The research 
thus limited itself to a method to calculate an optimal 
solution and in some cases involved a sensitivity 
analysis. 

Table 3 shows that 21 of the 41 papers confirmed the 
(partial) acceptance of the recommendations. In 18 of 



Table 2 Results Section IV and V: Factors related to the 
technical quality and process quality of a simulation 
study 



Section IV: Technical quality factors 


Times 




cited 


Data availability 


57 


Validation and verification through historic data 


37 


Quality of the conceptual model 


31 


Validation and verification through expert opinion 


30 


Keep the model as simple as possible 


25 


Quality of data 


21 


Quality of data analysis 


21 


Qthers 


23 


Model includes all relevant aspects 18 


Sensitivity analysis 18 


Section V; process quaiity factors 


Times cited 


Total commitment and support from user/client 


21 


involvement 




Appropriate use of animation in the model 19 


Qthers 


16 


Communication between those involved 


13 


Well defined objectives and project scope 


12 


Complete the project within time 1 1 


Realistic expectations between client and modeler 


5 


Do not exceed the available budget 


2 



the 41 papers (44%) the survey reported at least partial 
implementation of the results; the denominator (41), 
includes the categories missing, don't know, and not 
relevant. Additionally, the authors of 7 papers reported 
that the implemented changes were evaluated: 3 times 
with a time series designs (all from the same 
researcher), 1 with a controlled before and after design, 
1 with a before and after design and 2 with partial 
before and after designs. Altogether the survey showed 
that in 9 papers the implementation proved the model 
to be correct, 5 other models were partially correct. 

Discussion 

The literature review and the survey showed respectively 
an 18% and 44% implementation rate, suggesting that 
actual implementation occurred more often than 
reported in literature. Also the quality of the research 
methods in these cases was higher in reality (17%) than 
was reported in the literature (3%). Likewise the survey 
showed that more models proved to be correct in prac- 
tice than in the literature. However, the survey reported 
that 14 models proved to be at least partially correct, 
while only 7 projects were evaluated with a before and 
after design. This casts some doubt on the reliability of 
the survey. It seems that some authors reported their 
model to be correct based on (subjective) reactions of 
the hospital. 

An explanation for the differences between literature 
and practice is that the majority of papers focused on 
the technical simulation design and not the contribution 
to hospital improvements. This might be related to the 
authors' affiliations; the majority of the papers (66 out 
of 89) included at least one member of mathematical, 
operations research, industrial engineering or economics 
research groups. These researchers may prefer to pub- 
lish technical modeling details. Consequently, most con- 
ditions for success were related to the technical quality 
of the model. 

Fone et al. [9] provided another explanation; due to 
the time pressure to publish "it is likely that many mod- 
eling studies are published before validation is complete 
and before implementation has been carried out (and 
assessed). " 

In addition, scientific publications on improvements 
achieved with simulation may be hampered because of 
the difficulty to draft a "ceteris paribus" design, to find 
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Table 3 Results of the electronic survey of the authors 



Question Yes 


Partially No Do not know 


Missing 


Not relevant for study 


Total 


The study results are accepted by the hospital 1 1 


10 1 4 


3 


12 


41 


The study results are executed 7 


11 5 6 


0 


12 


41 


mplementation proved the study to be correct? 9 


5 1 3 


11 


12 


41 



control sites and of many unpredictable interfering 
variables. 

Our study found that the majority of factors contri- 
buting to actual implementation concerned the technical 
quality of the simulation. Data availability was the most 
frequently mentioned factor (57 times). At least 43% of 
the involved hospitals had to generate new data to gain 
sufficient insight into their problem, this percentage 
could even be higher as 27% of the papers did not 
report on data collection. Data availability is important 
because the reliability of a simulation model is affected 
by the quality of the data used to calculate input distri- 
butions. Validation and verification of the models are 
essential to check the quality of the model (see Table 2). 

Our study found that the process management factors 
are related to managing the expectations of the hospital 
and the modeler (see Table 2). This finding is consistent 
with Robinson & Pidd [24]. The use of animation was 
mentioned 19 times as a means to simplify communica- 
tion between modeler and hospital. Animation, however, 
should be used carefully as it can distract staff from the 
model details [27]. Many of the process management 
factors are in agreement with Forsberg et al. [28], as 
they also identified cooperation, careful planning of the 
project and stakeholder and customer involvement as 
success factors for implementation of simulation models 
in healthcare. 

Our finding that only 9 of the 89 papers reported the 
use of their model in more than one setting is in line 
with Proudlove et al. [29]. An explanation is that the 
emphasis placed on working closely with the client, 
meaning the best model for one hospital may be inap- 
propriate for other hospitals [30]. Often researchers 
make detailed models to increase the statistical descrip- 
tive power, but this can hinder the demonstration of 
general principles [29]. Others [31] plead for more gen- 
eric models after identifying differences between specific 
and general models in healthcare. In emergency care, 
initiatives have been undertaken to develop generic 
models [32]. 

Research limitations 

There may be a selection bias in the paper selection 
process as papers may be found in medical, health ser- 
vices and operations management and-research 
domains. We feel, however, that the included 89 papers 
are a good reflection of the available literature in this 



field. As this study focused on the implementation of 
recommendations and found a limited number of 
papers, we did not exclude papers because of the jour- 
nals' impact factor or the quality of the models. 

Although non-scientific literature contains many 
examples of simulation models in healthcare [6], we did 
not include these since non-peer reviewed articles are 
not held to the same rigorous quality standards. Addi- 
tionally, it is difficult to systematically identify these 
publications [6]. Another limitation is the possible bias 
in respondents of the survey. We were only able to con- 
tact authors from 67 of the 89 papers. Furthermore, of 
the contacted authors related to the 67 papers, only 41 
responded. It is more likely that staffs still present were 
involved in implementation. 

Future research 

The relative advantage that an innovation (here simula- 
tion) has over other methods affects the uptake [33]. 
This paper found that implementation took place in 
44% of the studies, however actual evidence that simula- 
tion leads to improved hospital performance is limited. 
To increase the uptake of simulation, researchers should 
provide high quality evidence of improvements. To get 
these results published, (scientific) journals could ask 
their authors to state whether or not the findings were 
accepted and implemented, and whether there is evi- 
dence of an improvement. A step further would be to 
have journals encouraging authors to submit follow-up 
papers describing the implementation of the recommen- 
dations. Furthermore, examining popular literature (e.g. 
trade journals, news media, etc.) on this subject remains 
an item for further research [6]. 

More research into perceived success factors seems 
necessary. Because of the encountered differences 
between literature and practice, it would be interesting 
to examine whether the technical, process and outcome 
quality of implemented recommendations are higher 
than those of studies that were not implemented. 
Furthermore surveys among multiple hospital respon- 
dents, such as management and medical professionals, 
that examine organizational characteristics contributing 
to the implementation of model findings seems worthy 
of further research. 

This research was limited to simulation studies on 
operations management in hospitals. It would be inter- 
esting to extent the scope to other techniques to enable 
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researchers to select the most appropriate OR techni- 
ques for specific settings. The overview of OR techni- 
ques and their advantages and disadvantages recently 
pubUshed by the RIGHT project is an important contri- 
bution [4] because it also discusses when to apply a spe- 
cific technique and the required resources. In addition 
generalization of the methods and results needs further 
attention. It is relevant to identify a pool of generic 
approaches and to design a decision schedule for its use, 
involving the contingent factors relevant for the decision 
to embark on a specific simulation approach. 

Conclusions 

This study showed that implementing recommendations 
of simulation applications on operations management in 
hospitals does not occur frequently; literature reports an 
18% implementation rate and a survey among these 
researchers indicates a 44% implementation rate. Formal 
post implementation evaluations were hardly reported 
upon in literature; 1 study described a before-and-after 
design, 2 described partial before-and-after designs. 

The lack of reported implementation of simulation 
recommendations might be explained by 1) academic 
factors: the authors' affiliations to mathematical/techni- 
cal research groups, the time pressure to publish which 
is hampered by the long duration of implementation 
research and the difficulty to draft a "ceteris paribus" 
design, and 2) individual and organizational barriers 
that affect the development of the model and its 
implementation. 

To ensure a wider uptake of simulation models in 
hospitals more evidence of improvements, based on rig- 
orous evaluation methods, seems necessary. Modelers 
and their clients - in this case relevant users within hos- 
pitals such as physicians, nurses and managers- should 
pay more attention to the perceived success factors that 
affect the technical, process and outcome quality of the 
simulation. 

Perceived success factors regarding the technical qual- 
ity are data availability, validation and verification with 
historic data, validation and verification through 



(internal) expert opinion, and the development of the 
conceptual model to be used in the simulation. Client 
involvement is most important for quality of the devel- 
opment process. Presenting the results in an under- 
standable and attractive way has a large impact on the 
usefulness of the model as it affects acceptance and 
actual implementation. 

Methods 

A literature review on simulation applications in indivi- 
dual hospitals provided answers to all research objectives. 
Relevant literature was checked on: 1) the reported 
implementation of recommendations, 2) the evidence 
that the changes resulted in improvements and, 3) com- 
ments of the researchers on factors contributing to the 
implementation of the study recommendations. 

In addition to the literature review, an email survey 
was sent to the traceable contact persons of the 
included papers. 

Search strategy and inclusion criteria 

As we expect literature on simulation applications in 
individual hospitals to be found in healthcare and busi- 
ness literature we selected the Business Source Elite 
(BSE) and the PubMed database. Business Source Elite 
(BSE) covers 930 journals on business, management, 
economics, finance and related topics while PubMed 
contains biomedical literature from MEDLINE (biome- 
dical literature) and life science journals. We used medi- 
cal subject headings for the search strategy in PubMed, 
while for BSE we searched the abstracts for specific key 
words. 

Two reviewers read all the abstracts and separately 
used the criteria of Table 4 to select papers for inclu- 
sion. We only included stochastic simulation applica- 
tions that discussed operations management in 
individual hospitals and were published between 1997 
and 2008. In case of disagreement, the full paper was 
retrieved and together a decision was made. Addition- 
ally, papers reporting (partial) implementation of the 
simulated recommendations were investigated further, 



Table 4 Inclusion and exclusion criteria for abstracts 



Inclusion criteria 



Exclusion criteria 



• The paper discusses an application of simulation 

• The discussed simulation model is stochastic. The current state does not 
determine the next state. 

• Goal of the simulation is to improve patient flow/process design or 
efficiency and resource capacity planning of primary processes (= processes 
related to patient care) 

• The simulation is concerned with processes within hospitals 



■ Other models than simulation 

■ Deterministic simulation models 

• Applications outside the hospital 

• Simulations concerning processes in hospital systems in which 
multiple hospitals collaborate 

• Simulation models that support medical decision-making (related 
to guidelines), or preventing errors related to the treatment 

• No surveys and reviews 

• No staff rostering 

• Papers published before 1997 

Papers written in other languages than English 
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Figure 2 Phases from simulation to improvement 



Improvement 



with all cross references included according to the cri- 
teria listed in Table 4. 

Data extraction 

Using literature, a scoring list with multiple-choice 
answers was developed to analyze all papers. In a pilot, 
consisting of 10 papers, the form was adapted to its the 
final form. 

The scoring list consisted of six sections. Section I 
asked for the simulated departments, the affiliation of 
the authors and the number of settings in which the 
model was used. 

Section II evaluated the extent of the implementation 
of the simulation. Figure 2 describes the implementation 
phases. The evaluation of the extent of the implementa- 
tion was based on a four stage model of success in 
simulation studies [24]. We separated the first stage (the 
study achieves its objectives and/or shows a benefit) 
into two separate phases: project achieves objectives and 
project results show benefits to client. We consider this 
a necessary distinction since we expect projects that 
achieve their objectives, but do not show benefits to the 
client, will not be implemented. 

Section III assessed the evidence that simulation leads 
to improvements (in case of implemented results) with 
the work of Eccles et al. [26] . It also examined whether 
implementation proved the results of the simulation to 
be correct. 

Section IV, V and VI, concerned quality factors of the 
simulation study as stated by the author. We considered 
three quality aspects: technical quality, the process in 
which the model is developed, and outcome quality [34]. 
The latter concerns the usefulness of the simulation; did 
it support the decision making process? The factors 
related to each quality aspect were based on Robinson 
& Pidd [24]. 

Two reviewers evaluated each paper separately and in 
case of discrepancies, they jointly determined the scores; 
in case of disagreement the third author was involved. 

Survey 

We were able to contact 67 of the 89 authors by email, 
the missing authors could not be traced. The survey 
consisted of 4 multiple choice questions that are com- 
parable to the data extraction form. The survey asked 
the authors 1) whether the hospital accepted the 



recommendations of the simulation study 2) whether 
the recommendations were implemented, 3) whether 
the impact of the implementation was evaluated 4) 
whether the simulation study proved the recommenda- 
tions to be correct. 
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